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Method and Apparatus for 3D Representation 

The present invention relates to a method and apparatus for deriving a three- 
dimensional representation from two or more two-dimensional images. 

As is well known, in general such a 3D representation can only be generated if the 
features (eg points) of the images are correlated. Respective features of two 
overlapping images are correlated if they are derived from (ie conjugate with) the 
same feature (eg a point) of the object. If the positions and orientations of the 
camera at which the overlapping images are acquired are known, then the 3D 
coordinates of the object in the region of overlap can be determined, assuming that 
the camera geometry (in particular its focal length) is known. 

Hu et al "Matching Point Features with ordered Geometric, Rigidity and Disparity 
Constraints" IEEE Transactions on Pattern Analysis and Machine Intelligence Vol 
16 No 10, 1994 ppl04 1-1049 (and references cited therein) discloses suitable 
algorithms for correlating features of overlapping images. Such algorithms can be 
used in the present invention and will not be discussed further. However it should be 
noted that in some circumstances (eg when the object is relatively simple or one of 
the known algorithms has failed to correlate sufficient features to enable an accurate 
reconstruction of part of an object to be derived) the features of the overlapping 
images can be correlated by eye and the correlation recorded on screen when using 
an embodiment of the method and apparatus of the present invention. 

Assuming that the correlated points of the images have been derived, in principle the 
information about the position and orientation of the camera required to reconstruct 
the object in 3D can be obtained either by direct measurement or by various 
sophisticated mathematical techniques involving processing all the correlated pairs 
of features. (Stereoscopic camera arrangements in which the cameras are fixed are 
ignored for the purposes of the present discussion). 

EP-A-782,100 discloses a method and apparatus in the first category, namely a 
photographic 3D acquisition arrangement in which a camera displacement signal 
and a lens position signal are used to enable a 3D representation of an object to be 
built up from 2D images acquired by a digital camera. Both the position and the 
orientation of the camera are monitored. The hardware required to achieve this is 
expected to be expensive. 

A number of research papers have addressed the more difficult question of deriving 
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the correct orientation from the correlated features of the images, as follows: 

i) E H Thompson "A rational Algebraic Formulation of the problem of Relative 
Orientation" Photogrammetric Record Vol 3 No 14 (1959) pp 152-159 sets out a 
5 mathematical procedure for aligning two images of the same scene to the correct 
orientation for reconstructing the scene in 3D, involving an iterative solution of five 
simultaneous equations derived from five pairs of correlated points. 
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ii) Richard Hartley et al "Stereo from uncalibrated cameras" Proc. IEEE Conf 
Computer Vision and Pattern Recognition (1992) pp 761-763 teaches that two 3x4 
camera matrices define the camera orientations and locations as well as the internal 
camera parameters such as focal length. If the cameras are calibrated (ie the internal 
parameters are known) then the camera matrices can be found from the matched 
points and hence the true 3D locations of all the points can be found. If the cameras 
are not calibrated then known "ground control" points must be used to derive the 
camera matrices and hence the 3D locations of the matched points. 

iii) Richard Hartley "Estimation of Relative Camera Positions for Uncalibrated 
Cameras" Computer Vision-ECCV'92, LNCS-Series, Vol 588, 1992 pp 579-587 is a 
development of the above Hartley paper and shows that the focal lengths as well as 
the positions and orientations of the cameras can be found from the matched points 
if all the other internal camera parameters are known. 

iv) Hartley et al "Computing Matched-epipolar projections" Computer Vision and 
25 Pattern Recognition 1993, pp 549-555 is a development of the above which 

introduces the epipolar transformation matrix to transform the images to images that 
would be acquired by cameras placed side-by-side with their optical axes parallel. 
The remaining points can then be correlated more easily. Also the cameras do not 
need to be calibrated. 



20 



30 



35 



A disadvantage of the above mathematical methods is the intensive computation 
required to process all the correlated points. Much of this computation involves the 
determination of the camera positions and orientations from multiple points and is 
effectively wasted because the positions and orientations are grossly over- 
determined by the large number of processed points. 

It has been shown that in general all the information on the viewpoints (ie position 
and orientation) of the camera(s) can be found from eight pairs of correlated points. 
However there are special cases in which this is not possible. One extreme example 
is a situation in which all eight points on the object corresponding to the pairs of 
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corelated points in the images are colinear. 

Importantly, another situation in which the camera positions cannot be determined is 
when the camera orientations (in the reference frame of the object) are identical. As 
will be apparent from Figure 3 (discusssed below) the camera separation in such a 
situation is quite indeterminate, no matter how many pairs of points are correlated. 
In such a situation the size of the object cannot be determined from the two images, 
even if the focal length and other camera parameters are known, and the 3D 
reconstruction will need to be multiplied by a scaling factor. 

Bearing in mind that the resolution of many digital cameras is severely limited (eg to 
640 x 480 pixels) it will be apparent that if the camera orientations are only slightly 
different then there will be considerable uncertainty in the camera positions and 
hence in the 3D reconstruction of the object. On the other hand if the camera 
orientations are deliberately chosen to be very different (ie converging sharply on 
the object) then points on the object in overhanging regions which are in the field of 
view when one image is acquired will not be in the field of view when the other 
image is acquired, preventing 3D reconstruction of regions of overhang. 

An object of the present invention is to overcome or alleviate at least some 
disadvantages of the known methods and apparatus, particularly when the resolution 
of the images from which the 3D reconstruction is generated is limited 

Accordingly, in one aspect the invention provides a method of processing 
overlapping 2D images of an object acquired from different spaced apart viewpoints 
relative to the object, the separation between the viewpoints not being precisely 
known, the 2D images being associated with pairs of correlated features, each such 
pair of correlated features comprising a feature of one 2D image and a feature of 
another 2D image derived from a common feature of the object, there being a 
mutual offset between the features of each pair which is determined by the relative 
rotation (if any) and translation between the viewpoints, the method comprising the 
step of digitally processing the 2D images to form a 3D reconstruction in which said 
common feature of the object is located in a simulated 3D space in dependence upon 
both said mutual offset and a scaling variable which is preferably entered manually 
by the user. 

For example if the actual camera separation is a and the partial 3D reconstruction is 
generated by virtual projectors with the same optical parameters as the camera(s) 
and having a separation of a' in simulated 3D space then the scaling factor could be 
a/a' to magnify the partial 3D reconstruction by a factor of a/a* and thereby generate 
a partial 3D reconstruction having the same size as the object.. Such a partial 3D 
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reconstruction will be able to be fitted to other life-size partial 3D reconstructions 
generated similarly from other pairs of images. In these embodiments, any value of 
scaling factor which will enable the partial 3D reconstructions to be fitted together 
will be satisfactory, and can for example be applied by the user during a process of 
fitting together the partial 3D reconstructions on-screen. 

In these embodiments the camera orientation (in the reference frame of the object) at 
each of the two viewpoints is preferably the same or nearly the same (eg ± 10 
degrees) and the or each partial 3D reconstruction is preferably generated by virtual 
projectors having the same orientation as the camera(s) and having optical centres 
on the line joining the optical centres of the camera(s). 

In other embodiments the variable will correct for one or more distortions of the 
partial 3D reconstruction either laterally or in the depth direction (curvature of field) 
which, as shown below in connection with Figures 8 to 1 1 can arise from incorrect 
positioning of the virtual projectors eg a misalignment relative to the camera 
viewpoints. 

In related embodiments the partial 3D reconstruction will be distorted by 
delibarately misaligning one or both the virtual projectors relative to the camera 
viewpoints . 

Such a feature is useful in the fields of design, graphics and animation. 

Preferably a view of the reconstruction in the simulated 3D space is displayed on a 
screen and the variable is varied by the user in response to the view displayed on 
screen. 

Preferably the method further comprises the step of acquiring the overlapping 2D 
images from a camera which is moved relative to the object between the different 
viewpoints, the net movement of the camera between the viewpoints not being fully 
constrained. 

For example the camera could be mounted on a fixed slide so as to move 
transversely to its optical axis (so that its orientation and movement along two axes 
is constrained but its movement along the third axis is not) or it could be mounted on 
a tripod (so that its movement in the vertical direction and rotation about a 
horizontal axis are constrained). 



However in a preferred embodiment the camera is hand-held. 
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Optionally, the camera orientation can be measured with an inertial sensor eg a 
vibratory gyroscope and appropriate filtering and integrating circuitry as disclosed in 
our UK patent GB 2,292,605B which is hereby incorporated by reference. A 
Kalman filter is presently preferred for filtering the inertial sensor output signals. 

5 Preferably the orientation of the camera is varied after acquiring a first image from 
one viewpoint and before acquiring a second image from the other viewpoint so as 
to maintain its orientation relative to the reference frame of the object when 
acquiring the second image. 

1 ^ One purely optical way of ensuring that the orientation of the camera is unchanged 
relative to the orientation at the first viewpoint is to vary the orientation until the 
projections in the image plane of the camera of the correlated points of the first 
image and the corresponding points of the second image (which is preferably 
instantaneously displayed on screen) converge at a common point. In the special 

15 case in which the distance to the object from the image plane along the camera's 
optical axis is unchanged between the two viewpoints (in which case the above 
common point would be at infinity) then the orientation at the second viewpoint can 
be adjusted until the above projections are parallel. 

20 In another aspect the invention provides a method of determining the motion of a 
camera relative to an object in the field of view of the camera comprising the steps 
of projecting the paths of features of the image of the object to a common vanishing 
point and determining the orientation of the optical center of the camera lens in 
relation to this vanishing point. 

25 

In another aspect the invention provides apparatus for determining the motion of a 
camera relative to an object in the field of view of the camera comprising means for 
projecting the paths of features of the image of the object to a common vanishing 
point and means for determining the orientation of the optical center of the camera 
30 lens in relation to this vanishing point. 

Some difference in orientation can be tolerated and the resulting distortion in the 3D 
reconstruction subsequently corrected, as will be shown below. 

35 The smaller the difference in orientation, the smaller the distortion in the 3D 
reconstruction. Ideally the orientation of the camera is maintained unchanged 
between the two viewpoints. 



Preferably the angle subtended by a pair of correlated features at the corresponding 
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feature of the object is 90 degrees ± 30 degrees (more preferably ± 10 degrees). 
Ideally the subtended angle is exactly 90 degrees. This feature enables any distortion 
resulting from a slight change in orientation to be corrected more accurately. 

5 Following the partial reconstruction by the above method, complementary 3D 
reconstructions of different parts of the object obtained similarly from further sets 
of overlapping images can be fitted together. 

To the extent that the reconstruction is distorted then simple compensations for 
10 distortion parallel to the image plane and for curvature of field can be applied in 
order to enable the 3D reconstruction to be fitted. 

It is not necessary (and indeed in many cases it will be computationally inefficient) 
to correlate all possible features prior to determining the viewpoints of the camera(s) 
15 relative to the object. It will usually be simpler to derive the remaining correlations 
between features of the respective images once the viewpoints have been 
determined from a small number eg eight pairs of correlated features, by searching 
for further correlated features along epipolar lines determined from the viewpoint 
determination . 
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Once the correlations and positions and orientations of the cameras are known, the 
3D configuration of the object is obtainable by projecting each image from (eg a 
virtual) projector having the same focal length and viewpoint (position and 
orientation) as the camera which acquired that image. The principal rays from 
corresponding features of the respective images will intersect in (virtual) 3D space at 
the location of the object feature. 



Accordingly, in another aspect the invention provides a method of generating a 3D 
reconstruction of an object comprising the steps of projecting images of the object 
acquired by mutually aligned cameras into simulated 3D space from aligned virtual 
30 projectors, the separation of the virtual projectors being variable by the user. 



In another aspect the invention provides apparatus for generating a 3D 
reconstruction of an object comprising two mutually aligned virtual projector means 
arranged to project images of the object acquired by mutually aligned cameras into 
simulated 3D space, the separation of the virtual projectors being variable by the 
user. 

Preferably the difference in alignment of the virtual projectors (ie the angle between 
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them) is less than 45 degrees, more preferably less 30 degrees, and is most 
preferably less than 20 degrees, eg less than 10 degrees. Desirably this angle is less 
than 5 degrees, eg less than one degree. This feature enables overhanging features of 
the object to be captured from both camera viewpoints and also facilitates the 
determination of the line connecting the optical centres of the camera(s) at the 
different viewpoints as well as the correlation of features between the images. 

In yet another aspect the invention provides a method of deriving a 3D 
reconstruction of at least part of an object comprising the steps of acquiring 
overlapping 2D images of the object from two spaced apart viewpoints, correlating 
at least three pairs of features of the images in a region of overlap,digitally 
processing the images by projecting the images into virtual 3D space and varying 
the relative positions and/or orientations of the projections to ensure that the 
projections of the respective features of at least three of said correlated pairs 
intersect, and reconstructing a further portion of the object in the region of overlap 
by projecting further features of the 2D images from the resulting varied relative 
positions and orientations of projection. 

Preferably the origin of each projection is located on a line in simulated 3D space 
connecting the corresponding optical centres of the camera at the two viewpoints. 
As explained below with reference to Figure 3, this will result in a scaled partial 3D 
reconstruction of the object. 

In one embodiment a distortion parameter is entered by the user and applied to the 
3D reconstruction. For example, after projecting the overlapping 2D images from 
their nominal viewpoints such that the projections intersect to form an initial 3D 
reconstruction in simulated 3D space, the initial 3D reconstruction can be rotated 
whilst constraining the features of the initial 3D reconstruction which are generated 
from the intersecting projections of correlated features of the projected images to lie 
on the projections of those features from one of the 2D images, thereby forming a 
further 3D reconstruction. As is illustrated in Figure 9 (discussed below) this can be 
used to generate a reconstruction which is parallel to, and therefore a scaled replica 
of, the actual object surface. The above-mentioned rotation and constraint correct for 
the lateral distortion caused by the lack of parallelism of the initial 3D 
reconstruction and the 3D reconstruction generated from correctly aligned virtual 
projectors. 

However as noted above, in many applications it will be desirable to distort the 3D 
reconstruction relative to the original object in order to achieve a desired artistic 
effect 
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It should be noted that the aspect of the invention concerning the derivation of a 3D 
reconstruction is applicable to previously acquired images in conjunction with their 
associated pairs of correlated points, however acquired. 

Preferably the viewpoints are calculated from at least two (desirably at least three) 
pairs of correlated features and the 3D reconstruction of the object is generated in 
dependence upon said calculation of the viewpoints, the calculation of the 
viewpoints being performed on fewer than all derivable pairs of correlated features. 

This preferred feature of the invention is illustrated in Figure 4 (discussed below) 
which shows the derivation of the line connecting the viewpoints from the pencil of 
projections from three pairs of correlated features. Since the third projection from 
the third pair of correlated features P3 and P3' intersects the point VP already 
defined by the other two projections' intersection (and thus merely onfirms the 
unchanged orientation between the viewpoints) it is not strictly necessary to find the 
straight line joining the viewpoints. Hence this line can be found from just two pairs 
of correlated features if there is no change in camera orientation. However greater 
accuracy will be obtained if more than two pairs of correlated features are processed. 

Preferably said calculation is performed on fewer than one thousand pairs of 
correlated features, more preferably fewer than one hundred pairs of correlated 
features, desirably fewer than fifty pairs of correlated features eg eight or fewer 
pairs.. For example the calculation can be performed on four, three or two pairs of 
correlated points. 

Particularly when the viewpoints are parallel or nearly parallel (eg within 10 degrees 
of each other) the above preferred features result in a greater or lesser degree of 
economy of processing. 

In another aspect the invention provides image-processing apparatus for deriving a 
3D representation of at least part of an object from overlapping 2D images thereof 
acquired from different viewpoints and correlated features of the respective 2D 
images in a region of overlap which are derived from a feature of the object 
common to the respective 2D images, the apparatus comprising reconstruction 
means arranged to form a 3D reconstruction in which said common feature of the 
object is located in a simulated 3D space in dependence upon both the mutual offset 
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between said correlated features and avariable which in use is preferably entered by 
a user. 

Further preferred features of the invention are defined in the dependent claims. 

Preferred embodiments of the invention are described below by way of example 
only with reference to Figures 1 to 15 of the accompanying drawings, wherein: 

Figure 1 is a diagrammatic view of one apparatus in accordance with the invention; 

Figure 2 is a flow diagram of one method in accordance with the invention; 

Figure 3 is a ray diagram showing the relationship between the object, camera 
viewpoints, projector viewpoints and partial 3D reconstruction in one embodiment 
of the invention; 

Figure 4 is a ray diagram in 3D showing a derivation in accordance with one aspect 
of the invention of the direction of movement of the camera from the acquired 
images in the method of Figure 2 and apparatus of Figure 1; 

Figure 5 is a ray diagram in 3D showing a derivation of the direction of movement 
of the camera from the acquired images in the method of Figure 2 and apparatus of 
Figure 1 in the special case in which the camera does not move relative to the object 
in the Z direction; 

Figure 6 is a diagram showing the movement of the image of the object in the image 
plane I of Figure 5; 

Figure 7 is a flow diagram summarising the image processing steps utilised in t he 
method of Figure 2 and the apparatus of Figure 1; 

Figure 8 is a 2D ray diagram illustrating the curvature of field resulting from 
misalignment of one virtual projector relative to the other in the apparatus of figure 
1 and method of Figure 2; 

Figure 9 is a 2D ray diagram illustrating correction of distortion of the partial 3D 
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reconstruction in an embodiment of the invention; 

Figure 10A is a 2D ray diagram illustrating the curvature of field resulting from a 
misalignment of a virtual projector by 5 degrees; 

Figure 1 OB is a ray diagram illustrating the curvature of field resulting from a 
misalignment of a virtual projector by 10 degrees; 

Figure 10C is a ray diagram illustrating the curvature of field resulting from a 
10 misalignment of a virtual projector by 15 degrees; 

Figure 1 1 is a plot of curvature of field:misalignment in the arrangements of Figures 
10A to IOC; 
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Figure 10 is a diagrammatic perspective view illustrating the tracking of relative 
movement between an object and the camera by tracking 3 principal ray lines from 
a region of the object substantially parallel to the image plane of the camera; 



Figure 1 1 is a diagrammatic elevation showing the movement in 2D of the object as 
seen by the camera of Figure 10; 

Figure 12 is a view displayed by the screen of the apparatus of Figure 1 during the 
fitting together of two reconstructions of respective portions of the object; 
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Figure 13 is a view subsequently displayed by the screen of the apparatus of Figure 1 
during the fitting together of the 3D reconstruction of Figure 12 with a similarly 
obtained but relatively elongated partial 3D reconstruction of the object; 

Figure 14 is a view subsequently displayed by the screen of the apparatus of Figure 1 
illustrating the alternative compression in the vertical direction and compression in 
the horizontal direction of the resulting complete 3D reconstruction in order to 
achieve a desired relationship to the actual object, and 



Figure 15 is a diagrammatic representation of a screen shot generated by a program 
for manipulting the partial 3D reconstructions. 
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Referring to Figure 1, the apparatus comprises a personal computer 4 (eg a 

Pentium® PC) having conventional CPU, ROM, RAM and a hard drive and a 
connection at an input port to a digital camera 1 as well as a video output port 
connected to a screen 5 and conventional input ports connected to a keyboard and a 
mouse 6 or other pointing device. The hard drive is loaded with conventional 
operating system such as Windows ®95 and software: 

a) to display images acquired by the camera 1 ; 

b) to correlate points in overlapping regions of images input from the camera 1; 

c) to derive the line of movement of the camera from the acquired images; 

d) to project images acquired by the camera into a simulated 3D space from virtual 
projectors located on the line of movement at a separation selected (with the 
keyboard or pointing device) by the user (thereby creating a partial 3D 
reconstruction); 

e) to scale the partial 3D reconstruction along one or more axes and combine such 
partial 3D reconstructions as illustrated in Figures 12 and 13, and 

f) to apply or compensate for lateral distortion and curvature of field as illustrated in 
Figure 9 and Figures 10A to IOC. 

The software to carry out function a) can be any suitable graphics program and the 
software to carry out function b) can be based on the algorithms disclosed in Hu et 
al "Matching Point Features with ordered Geometric, Rigidity and Disparity 
Constraints" IEEE Transactions on Pattern Analysis and Machine Intelligence Vol 
16 No 10, 1994 ppl041-1049 (and references cited therein). One suitable algorithm 
is the Gruens algorithm. 

Referring to Figure 1, the camera 1, which may be hand-held and carrying a 3-axis 
vibratory gyroscope G with associated filtering circuitry (a Kalman filter is presently 
preferred) and integrating circuitry to generate and display on screen 3-axis 
orientation signals or (as shown at V) may be mounted on a tripod T or other 
support, for example, defines a first set of axes x, y, z in its initial viewpoint (ie 
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position and orientation) and is used to acquire an image of the object 3 and to 
display the image on screen 5. The origin of the x y z coordinate system is taken to 
be the optical centre of the camera lens. The camera is then moved an arbitrary 
distance to a new position V and a second image of object 3 is acquired with the 
orientation of the camera (relative to the x y z coordinate system) maintained 
unchanged. This unchanged orientation can be achieved with the aid of a suitable 
support if necessary. A method of checking for changes in orientation, based on the 
convergence of the projections from corresponding points of the two images in the 
image plane I of the camera, will subsequently be described with reference to Figure 
4. Only a few (eg 3, 4, 5 or up to eg 10) pairs of correlated points need to be found 
for this purpose and can be derived visually by the user from the images displayed 
on screen or by the software in computer 4. As will become apparent from Figure 4, 
the camera movement (ie the line in the x y z coordinate system joining the optical 
centre of the camera lens in its two positions) can also be found at his stage. In some 
cases however in which a highly accurate reconstruction is not required, the 
direction of camera movement can be estimated by the user. 

The remaining corresponding points in the two images are then correlated by the 
computer 4 (preferably taking advantage of the information on camera movement 
obtained in the previous step, eg by searching along the epipolar line in one image 
corresponding to a point of interest in the other image) and a partial 3D 
reconstruction of the object 3 in simulated 3D space is generated from the correlated 
points as will be described below with reference to Figure 3. Since the distance 
between the positions 1 and 1' is not known, a parameter representing this distance 
is entered by the user, either form the keyboard or from the pointing device 6 for 
example. The computer 4 is programmed to display the resulting 3D reconstruction 
on screen and the user can vary the parameter interactively in order to arrive at a 
partial 3D reconstruction in the x y z coordinate system which bears a desired 
relationship to the actual object 3. Typically this will be stretched or compressed in 
the direction of movement between positions 1 and 1% relative to the actual object. 

Depending on the accuracy with which the camera orientation is maintained in the 
two positions (eg with the aid of orientation signals from gyroscope G) and the 
accuracy of the estimated or optically derived camera movement there may be 
distortions in the partial 3D reconstruction which can be corrected at this stage. The 
correction of such distortion (and indeed the deliberate addition of such distortion 
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when this is desired) will be described subsequently with reference to Figures 9 and 
10A to IOC. 

A further partial 3D reconstruction is then generated by moving the camera 1 to a 
new viewpoint 1 A such that the object lies in a region of overlap ie such that at least 
one point P in the camera's field of view at viewpoints 1 and 1' remains in the 
camera's field of view at viewpoint 1A. This movement can be represented by a 

rotation 9 about the y axis (resulting in new axes xi, yi and zj) followed by a rotation 
(|> about the xj axis (resulting in new axes x', y' and z 9 ) followed by a rotation k 
about the z' axis, followed by translations AX, AY and AZ along the resulting axes. 
Therotation k about the z 9 axis will in many cases be zero, as shown in Figure 1. 

After acquiring an image at the viewpoint 1 A the camera is moved to a new position 
1A' and a further image is acquired and displayed (the orientation of the camera 
being adjusted to remain the same as at viewpoint 1A). A further partial 3D 
reconstruction of object 3 is then performed by the computer 4 in a manner 
analogous to that described above in connection with viewpoints 1 and 1\ 

If it is assumed that the origin of the x% y', z 9 coordinate system is shifted to the 
optical centre of the camera at viewpoint 1 A to give a new coordinate system X, Y, 
Z then the relationship between any point XYZ in the new XYZ coordinate system 
and the same point xyz in the xyz coordinate system is given by: 



r X\ ( soskcosG sinKcos(|) - cosKsin0sin<f> sinKsin(|> + cosKsinGcosa/ x \+ 



Y| = 



-sinKcosS cosKcos<j) + sinKsin0sin()> cosKsin<|> - sinKsinOcos<|> I y 
-sin0 -cos6sin<|> cos8cos(|> J\z J + 



The terms in the 3x3 matrix can be shown to be the cosines of the angles between 
the axes of the XYZ and xyz coordinate systems (see eg Boas "Mathematical 
Methods in the Physical Sciences" Pub John Wiley & Sons, 2nd Edn pp437 and 
438). 

Accordingly the partial 3D reconstructions can be transformed to a common 
coordinate system and elongated/compressed along all three axes to minimise 
discrepancies between them in their region of overlap and to build up a more 
complete reconstruction of the object 3. 
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In the present embodiment this process is carried out under the control of a user by 
displaying the partial 3D reconstructions on screen and varying their relative 
elongation/compression along three axes, as will be described in ore detail with 
reference to Figure 13. 

In another embodiment the partial 3D reconstructions are combined without user 
intervention using the Iterative Closest Point algorithm. This algorithm is publicly 
available and therefore it is not necessary to describe it in detail. Briefly however, it 
registers two surface edges by searching for a group of (say) the ten closest pairs of 
points on the respective edges. The surfaces are then repositioned to minimise the 
aggregate distance between these pairs of points and then a new group of closest 
pairs of points is found. The repositioning step is then repeated. Other methods of 
correlating 3D surface regions are disclosed in our GB 2,292,605B (supra). In 
accordance with the present invention, a scaling factor or other variable is generated 
either by the user or iteratively under software control in order to adjust the relative 
sizes of the partial 3D reconstructions to ensure they can be fitted together into a 
self-consistent overall surface description of the object. 

Returning to the description of the embodiment of Figures 1 and 2, further partial 3D 
reconstructions can be derived similarly from the other sides of the object 3 and 
combined with each other and/or with the existing combination of partial 3D 
reconstructions until a complete 3D representation of the object is achieved. 

The method is summarised in Figure 2. 

Overlapping images are captured (step S10), pairs of points (or larger features eg 
lines or areas) are correlated (step S20) and at least the approximate camera 
movement between the viewpoints is determined (step S30). In the preferred 
embodiment this is determined with a high degree of accuracy by processing the two 
images. Partial 3D reconstructions of the object surface are then generated in a 
simulated 3D space using the computer 4 to process the correlated pairs of points 
and camera movement (step S40) and these are combined, preferably interactively 
on screen by the user to give a consistent but possibly distorted (eg compressed or 
elongated) 3D representation (step S50). Optionally, this is distorted or undistorted 
by applying appropriate compression or elongation (step S60). 
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In an alternative embodiment, steps S30 and S40 can be combined in a matrix 
processing method, eg using the commercially available INTERSECT program 
produced by 3D Construction Inc. 

Step S40 will now be described in more detail with reference to Figure 3. 

Figure 3 shows the object 3 in the field of view of the camera 1 at positions 1 and 1\ 
Principal ray lines from points PI, P2 and P3 on the surface of the object pass 
through optical centres Ol and 02 of the camera in the respective positions 1 and V 
and are each imaged on the image plane. The pair of points thus formed on the 
image plane from each of points PI, P2 and P3 (and many other points, not shown) 
are corresponding points and can be correlated by known algorithms. 

Accordingly it will be appreciated that if virtual projectors PR1 and PR2 with the 
same orientation, focal length and other optical characteristics as the camera at its 
respective viewpoints were placed at positions 1 and V then they would project ray 
lines into virtual 3D space from the correlated pairs of points which would intersect 
at the true locations of the corresponding points PI, P2, P3... and all other points on 
the object surface corresponding to a correlated pair of image points. By a virtual 
projector is meant any operator on the image which behaves as an optical projector 
of the image. A suitable software routine for processing the image in the required 
manner could be written without difficulty be persons skilled in the art. 

Although the projectors are shown in Figure 3 with the same orientation in the 
reference frame of the object (corresponding to the common camera orientation) this 
is merely a preferred feature which enables the direction of the line ST to be 
determined more easily by the image processing method disclosed in Figure 4. In 
principle the above analysis is also applicable to virtual projectors of different 
orientations, corresponding to respective, different camera orientations. As noted 
above, the camera orientations can be determined by an inertial sensor and 
integrating circuitry similar to that disclosed in our above-mentioned UK patent GB 
2,292,605B. 

If the line ST joining the optical centres of the camera/projector lenses in positions 1 
and 1 * is extended and projector PR2 is moved along this line with its orientation 
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unchanged (eg to position PR2' as shown) and the same ray lines are projected from 
it then these will still intersect the ray lines from projector PR1 but at different 
positions in simulated 3D space. For example the intersections at PI, P2 and P3 will 
be replaced by intersections at Ql, Q2 and Q3 respectively. This is because any ray 
line from 03 parallel to a ray line from 02 will lie in the same plane as a ray line 
from Ol intersecting the ray line from 02 and therefore intersect that ray line from 
Ol. For example line 03Q1 lies in the same plane as triangle 01P102 and will 
therefore intersect line OlPl, in this case at Ql. 

Hence a scaled representation 30 can be generated by any pair of virtual projectors 
on line ST, if line ST is parallel to the line joining the optical centres of the camera 
lens at the two positions at which the projected images are acquired and the 
projectors have the same orientation(s) as the camera. This last condition can be 
satisfied even if the correct orientation is initially unknown, namely by adjusting the 
orientation about any axis perpendicular to line ST until the respective ray lines 
from any pair of correlated points intersect. This procedure can be carried out either 
manually be the user with the aid of a suitable display of the images and ray lines on 
screen or automatically in software. 

Accordingly a 3D representation of the object can be generated once the direction of 
line ST is known. 

Figure 4 illustrates one derivation of line ST (step S40 in Figure 2). For ease of 
understanding, the camera (having a lens with optical centre O) is considered to be 
stationary and the object 3 is considered to move to position 3' along line M. Points 
PI, P2 and P3 are imaged as points pi, p2 and p3 when the object is in position 3 
and as points pl\ p2' and p3' when the object is in position 3'. If and only if the 
orientation of the camera relative to the object frame is unchanged between the two 
positions, then the projections of the lines LI, L2 and L3 joining Pn and pn' (n = 1 
to 3 in this illustration but in general will be much larger) will meet at a common 
point VP, which is somewhat akin to the vanishing point in perspective drawing. 

It will be seen that the line joining VP to the optical centre O is the line ST which is 
the locus of the optical centre of the camera (or desired locus of the virtual 
projector) relative to the object 
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In general, if the camera is hand-held, the lines LI, L2, L3....Ln connecting the 
correlated points will not meet at a common point, owing to a change in orientation 
of the camera between the two positions. However the orientation of the camera can 
be varied about the X, Y and Z axes by the user at the second position whilst 
displaying the lines LI, L2, L3...Ln on screen and the second image captured only 
when these lines converge on or near a common point, as determined either visually 
or by a software routine. Indeed the image can be captured, under control of the 
software routine, only when the necessary convergence is achieved. 

If gyroscope G is used then its 3-axis orientation signals can be used to maintain the 
orientation of the camera between the two viewpoints. 

It should be noted that if the camera is considered to be moving and the object to be 
fixed, then the movement of the camera from its original position is derived by 
superimposing the image plane and associated image of the camera in its new 
position on the image plane (and image) of the camera in its first position, projecting 
the lines LI, L2, L3....Ln in the image plane of the camera in its first position and 
connecting the resulting point of intersection VP to the optical centre of the camera 
in its first position. The resulting line ST is the movement of the object in the 
coordinate frame of the camera at its first position. 

In a variant of the above method, a moving image or a rapid succession of still 
images are acquired by the camera as it moves and the (assumed) rectilinear 
movement of the camera between each still image or between sequential frames of 
the moving image is derived by projecting the lines LI, L2, L3....Ln in the image 
plane of the camera to derive the point VP for each incremental movement of the 
camera. The resulting line segment ST for each incremental movement of the 
camera will change direction as the direction of movement of the camera moves and 
the segments can be integrated to determine the overall movement (including any 
change in orientation) of the camera. 

It should be noted that when there is little or no movement of the camera relative to 
the object in the depth (Z) direction then lines LI, L2, L3...Ln will be parallel or 
nearly so and VP will be at infinity or too far away to be determined with accuracy. 

This special case is illustrated in Figures 5 and 6. Points A, B and C at corners of 



• # 
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object 3 in its initial position are imaged by lens L as triangle abc in the image plane 
I of the camera. When the object moves to a new position 3' these comers A', B' 
and C are imaged as triangle a'b'c' which will be similar (in the narrow geometrical 
sense) to triangle abc but may for example be rotated. For reasons whch are 
explained below, it will be assumed initiallly that points A, B and C lie in a plane 
which is substantially parallel to the image plane I. 

It should be noted that the images abc and a'b'c' would also be obtained from a 
smaller object of the same shape correspondingly nearer the camera, as illustrated by 
faceAlBlCl. 

Referring now to Figure 6 which shows the object 3/3' (not its image) as seen by the 
camera, various possible faces ABC, A1B1C1 and A2B2C2 are shown. There will 
be a continuous range of possible sizes for face ABC; for the sake of clarity only the 
above three are shown. However the possible faces all have a common centroid P. 

When the object moves to a new position 3' illustrated by face A'B'C the centroid 
will move to a new position Q and the line PQ, which will be parallel to the image 
plane I (Figure 5) will represent the direction of movement irrespective of the true 
size and distance of the object. In fact this will remain true even if there is some 
rotation about the Z axis (such that the lines AA\ BB' and CC are not parallel). 

However if the camera is rotated at the second position about the Z axis to ensure 
that the above lines AA\ BB* and CC are in fact parallel, then the above analysis 
holds true even if the points A, B and C do not lie in a plane parallel to the image 
plane I; for example if the image abc is derived from points A1BC in Figure 5. The 
corresponding centroid (not shown) of A1BC in Figure 6 is displaced from centroid 
P and the corresponding centroid of Al'B'C after movement of the object to 
position 3' is similarly displaced from centroid Q. However the line joining these 
new centroids will be parallel to line PQ and will therefore correctly indicate the 
direction of movement of the object relative to the camera. 

Even if there is some movement of the camera in the Z direction such that lines AA\ 
BB' and CC are not parallel but converge to a distant point, line PQ will still 
correctly represent the direction of movement of the object 3. 
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Thus the above method overlaps to some extent with the method of Figure 4. 

The overall method of determining the direction of movement of the camera is 
illustrated in Figure 7. A first image is captured (step SI 1) and the camera is moved 

5 to a new position with the object still in its field of view and the first image is 
displayed on screen 5 (Figure 1), superimposed on the instantaneous image seen by 
the camera at the new position (step S12). Corresponding points are correlated, 
either by eye (ie by the user) or by a suitable software routine (step S13). Only a 
small number of points need to be correlated at this stage, eg 100 or fewer, eg 10, 

10 depending on the processing speed and the accuracy required. 

If it appears to the user that there has been movement in the Z direction the method 
branches to step S14 at which the orientation of the camera is adjusted about the X, 
15 Y and Z axes until the correlated points converge to a common "vanishing point" 
VP (Figure 4). At this stage the second image is captured and stored in memory. 

In step SI 5 a line is projected from the point VP through the optical centre of the 
camera lens to find the locus ST (Figure 4) and with the aid of this information, 
20 further correlation of the images is performed and a partial 3D reconstruction is 
generated by the moethod illustrated in Figure 3 (step S17). 

Preferably, step S15 also involves or is preceded by the selection of a camera model 
(eg from a list displayed on screen) by the user. The computer 4 is programmed to 

25 

store a list of camera models Ml, M2....Mn in association with the parameters of 
each model needed for the calculation of the required projection and other 
processing.The following parameters may be stored: 

30 Camera model Mn: 

pixel size in x direction 
pixel size in y direction 
focal length 
35 x dimension of film plane (in pixels) 
y dimesnion of film plane (in pixels). 



If it appears to the user (eg as a result of a failure to find a reasonably close 
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"vanishing point'* VP) that there has not been significant movement in the Z 
direction then the movement of the object is assumed to be parallel to the image 
plane and the line PQ is found from the images of a group of eg three or more points 
(not necessarily corners) preferably lying in or close to a plane parallel to the image 
5 plane (step SI 6) before proceeding to step S17. 

If only because most digital cameras currently on the market have a limited 
resolution (eg 1024 x 768 pixels, 640 x 480 pixels or even less) there will almost 
inevitably be some error in the determination of the camera movement and hence in 
10 the relative placement and orientation of the virtual projectors PR1 and PR2 in 
Figure 3. The effect of this error is illustrated in Figure 8. 

Figure 8 is a ray diagram orthogonal to the image plane showing the imaging of a 
line of points PI, P2 and P3 on the surface of the object 3 by a camera at positions 1 
and 1\ For the sake of clarity the lens L is shown only at the first position 1. It will 
be noted that the orientation of the camera is the same at the two positions. 
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Virtual projectors PR1 and PR2 would correctly reconstruct the object as 
reconstruction 30 if located at the above positions with the above orientation. If 
projector PR2 were moved to position PR2' along the direction of movement ST 
(Figure 4) of the cameras then the resulting reconstruction 30' would still show the 
points (Pl\ P2' and P3') on a straight line with no curvature of field,with correct 
scaling in the lateral direction (ie the length ratio P1P2/P2P3 = PrP27P2*P3'). 

If the projector is successively rotated to a new positions PR2" and PR2*" then 
successively greater curvature is introduced into the reconstruction 30" or 30'", as 
shown. The relationship between the above curvature of field of the reconstruction 
30 and the angular error in the orientation of the second virtual projector relative to 
the first is considered below in relation to Figures 10A to IOC. 



However, referring to Figure 9, in which the respective ray lines from the correlated 
points of the images projected by projectors PR1 amd PR2 are substantially 
35 orthogonal, it can be seen that an error in the relative orientation of the projectors, 
resulting in the projection of the second image from projector PR2\ results in a 
negligible curvature of field: line 30' defined by the intersection of the ray lines 
from PR2' with the corresponding ray lines from projector PR1 is substantially 
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straight, like line 30 which is defined by the corresponding intersections of the ray 
lines from correctly co-oriented projectors PR1 and PR2. Accordingly it is a 
preferred feature of the invention that the camera positions are so chosen that the 
angle subtended by the ray lines from a pair of correlated points at their intersection 
at their corresponding point at the object surface is substantially 90 degrees, eg 90 
degrees ± 30 degrees, preferably 90 degrees ± 20 degrees, most preferably 90 
degrees ±10 degrees. This minimises the correction required due to curvature of 
field. 

There remains the lateral distortion of the reconstruction, ie the discrepancy between 
the ratios P1P2/P2P3 and PrP27P2'P3' (Figure 8). If however the partial 3D 
reconstruction 30' is rotated to position 30C so as to lie parallel to the image plane 
of projector PR1, as shown in Figure 9, and all its points are shifted so as to to be 
intercepted by the ray lines from projector PR1, then the triangles defined by these 
shifted points and the ray lines from PR1 will be geometrically similar to the 
triangles defined by the corresponding points on line 30 and these ray lines, and 
hence the distribution of points along line 30C will be a scaled replica of the 
corresponding distribution along line 30. It will be understood that this is so 
irrespective of the angle subtended by the ray lines at their intersection at their 
corresponding point at the object surface. 

Accordingly, distortion parallel to the image plane can be corrected for (or 
deliberately applied) by rotating the partial 3D reconstruction about an axis 
perpendicular to the image plane of a projector used to generate that partial 
reconstruction whilst constraining the points in the partial 3D reconstruction to lie 
on their ray lines from that projector. 

Figures 10A to 10C illustrate the curvature of field resulting from angular 
misalignment of one projector PR2' from its correct orientation PR2. In each case 
the correct reconstruction 30 defined by PR1 and PR2 is planar and the centre of 
curvature CN of the actual reconstruction 30' is shown (and was derived 
geometrically). In Figures 10A to 10C, the misalignment is generated by rotation of 
the projector PR2 about the optical centre of its lens by 5 degrees, 10 degrees and 15 
degrees respectively. 

The radius of curvature R of reconstruction 30' is inversely proportional to the 
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misalignment as shown in Figure 1 1. It should be noted that in Figures 10A to IOC, 
the angle subtended by the ray lines at their intersection at their corresponding point 
at (the reconstruction 30 of) the object surface is very much less than the optimum 
angle of 90 degrees and hence the degree of curvature is very much greater than 
would normally be obtained in practice. 

Hence the method of the invention allows a considerable latitude in the orientation 
of the projectors which implies that a considerable uncertainty in the relative 
orientation of the camera at the two viewpoints is permissible. 

The corrections noted above can be applied at any stage during the generation or 
fitting together of the partial 3D reconstructions. 

It should be noted however that only a limited misalignment of the virtual projectors 
PR1 and PR2 (Figure 3) is permissible. Referring to Figure 3, all the principal ray 
lines projected from the correlated points will intersect only if the projectors have 
the correct orientation. The latitude in orientation arises only from the finite 
resolution of the camera(s) which implies that the ray lines from the projectors have 
20 a finite thickness or do not intersect exactly. Thus a point on the reconstruction 30 
can be found by determining the midpoint of the shortest line joining the ray lines 
from the respective projectors, provided that the length of this shortest line does not 
exceed a predetrmined limit corresponding to the resolution of the camera(s). 



Figures 12 to 14 illustrate the fitting together and manipulation of the partial 3D 
reconstructions on screen. 

In Figure 12 a partial 3D reconstruction Rl and a partial 3D reconstruction R2 are 
fitted together at their corresponding points F and F\ A and A' and result in a 
composite (but still incomplete) 3D reconstruction R3 as shown in Figure 13 having 
an array comprising points E, B and D from Rl and points C and G' in defined 
positions which can be fitted to a further partial 3D reconstruction R4 as shown in 
Figure 13. 

Referring to Figure 13, it is assumed that faces ABC, DGCB, DGEF and ABDEF of 
R4 are formed but that the region bounded by points A, C , G and F is not. The user 
rotates R3 on screen using eg mouse 6 (Figure 1) to give a rotated partial 3D 
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representation R3' which is aligned with R4 and which he/she then sees is 
horizontally compressed relative to R4. Accordingly R3 can be expanded 
horizontally or R4 can be compressed horizontally to enable them to be fitted 
together to form a self consistent, complete 3D representation of the object R5. 

As shown in Figure 14, this can be compressed or expanded along any desired axis 
or otherwise distorted under the control of the user to produce a final representation 
R6 or R6* having any desired relationship with the actual object 3. 

Figure 15 shows a screen shot of the user interface generated by a program for 
manipulating the 3D reconstructions 30. Buttons BN are provided which can be 
clicked on by the mouse under the control of the user and,when thus selected, enable 
the user to drag portions of the displayed representation so as to zoom, rotate and 
panthe view of the object, as well as tocome closer to and move away from the 
object ("nearer" and "further" buttons respectively). As described this far, the 
interface is similar to the publicly available interface of the COSMO PLAYER web 
browser plug-in produced by CosmoSoftware Inc, of California USA. However, in 
accordance with a feature of the present invention, "wheels" Wl and W2 are 
provided which are rtoatable by the mouse and enable the user to adjust the 
separation between the virtual projectors and to vary the distance to the object 
respectively. The latter control is effectively a perspective control. Optionally, 
further buttons or other means (not shown) may be provided to enable distortions 
(such as thos described above) to be applied in a graphical fashion, or to enable 
other distortions such as shear dostortion to be applied. 

Preferred features of any aspect of the invention can be combined with any other 
aspect of the invention. 
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Claims 

1. A method of processing overlapping 2D images of an object acquired from 
different spaced apart viewpoints relative to the object, the separation between the 
viewpoints not being precisely known, the 2D images having pairs of correlated 
features, each such pair of correlated features comprising a feature of one 2D image 
and a feature of another 2D image derived from a common feature of the object, 
there being a mutual offset between the features of each pair which is determined by 
the relative rotation (if any) and translation between the viewpoints, the method 
comprising the step ofdigitally processing the 2D images to form a 3D 
reconstruction in which said common feature of the object is located in a simulated 
3D space in dependence upon both said mutual offset and ascaling variable . 

2. A method as claimed in claim 1 wherein a view of the reconstruction in the 
simulated 3D space is displayed on a screen and the scaling variable is entered by 
the user. 

3. A method as claimed in claim 1 or claim 2 further comprising the step of 
20 acquiring the overlapping 2D images from a camera which is moved relative to the 

object between the different viewpoints, the net movement of the camera between 
the viewpoints not being fully constrained. 



4. A method as claimed in claim 3 wherein the respective orientations of the camera 
at the different viewpoints relative to a reference frame which is fixed with respect 
to the object differ by less than 45 degrees. 

5. A method as claimed in claim 4 wherein the difference between said respective 
orientations of the camera is less than 30 degrees. 

6. A method as claimed in claim 5 wherein the difference between said respective 
orientations of the camera is less than 10 degrees. 

35 7. A method as claimed in any preceding claim wherein the 3D reconstruction is 
generated by projecting the correlated features from positions on a straight line in 
simulated 3D space which corresponds to the straight line joining the positions of 
the camera at which said 2D images are acquired. 
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8. A method as claimed in claim 7 wherein said straight line in simulated 3D space is 
determined from a pencil of projections from correlated pairs of points of aligned 
2D images. 

9. A method as claimed in claim 7 wherein said straight line in simulated 3D space is 
determined from parallel projections from correlated pairs of points of aligned 2D 
images. 

10. A method as claimed in any preceding claim wherein the camera orientation is 
determined from an output signal of an inertial sensor. 

11. A method as claimed in any of claims 3 to 10 wherein the 2D images are 
acquired by a hand-held camera. 

12. A method as claimed in any preceding claim wherein said scaling variable is 
varied by the user to enable the 3D reconstruction to be fitted to another 3D 
reconstruction similarly derived from 2D images acquired at another pair of 
viewpoints. 

13. A method as claimed in any preceding claim wherein said features are points. 

14. A method as claimed in any of claims 1 to 12 wherein said features are lines. 

15. A method as claimed in any of claims 1 to 12 wherein said features are areas. 

16. A method as claimed in any of claims 3 to 15 wherein the viewpoints are 
calculated from one such mutual offset between a first pair of correlated features at 
and a further such mutual offset between a second pair of correlated features, the 
calculation being performed on fewer than allderivable pairs of correlated features, 
further pairs of correlated features are derived from a search along epipolar lines 
derived from the viewpoint determination and the 3D reconstruction of the object is 
generated from the viewpoints and further pairs of correlated features. 

17. A method as claimed in claim 16 wherein said calculation is performed on fewer 
than one thousand pairs of correlated features. 
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18. A method as claimed in claim 17 wherein said calculation is performed on fewer 
than one hundred pairs of correlated features. 

19. A method as claimed in claim 18 wherein said calculation is performed on fewer 
than fifty pairs of correlated features. 

20. A method as claimed in claim 19 wherein said calculation is performed on four 
eight or fewer pairs of correlated points. 

21. A method as claimed in claim 20 wherein said calculation is performed on two 
or three or four pairs of correlated points. 

22. A method as claimed in any preceding claim comprising the step of repeating the 
method of claim 1 by electronically processing further overlapping 2D images of the 
object acquired from different further viewpoints to form a further 3D 
reconstruction, the first-mentioned 3D reconstruction and the further 3D 
reconstruction being combined by manipulations in a simulated 3D space involving 
one or more of rotation and translation, any remaining discrepancies between the 3D 
reconstructions being reduced or eliminated by scaling one 3D reconstruction 
relative to the other along at least one axis. 

23. A method as claimed in claim 22 wherein at least two of the 3D reconstructions 
are simultaneously displayed on screen and their manipulations are performed in 
response to commands entered by a user. 

24. A method as claimed in claim 23 wherein the manipulations of the 3D 
reconstructions are performed under the control of a computer pointing device 
operated by the user. 

25. A method as claimed in any of claims 22 to 24 wherein a distortion parameter is 
entered by the user and applied to said first-mentioned and/or said further 3D 
reconstruction. 



26. A method as claimed in claim 25 wherein overlapping 2D images are projected 
from their nominal viewpoints such that the projections intersect to form an initial 
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3D reconstruction in simulated 3D space, and the initial 3D reconstruction is rotated 
whilst constraining the features of the initial 3D reconstruction which are generated 
from the intersecting projections of correlated features of the projected images to lie 
on the projections of those features from one of the 2D images, thereby forming a 
5 further 3D reconstruction. 

27. A method as claimed in any of claims 22 to 26 wherein a further parameter 
indicative of curvature of field is entered by the user and used to adjust the curvature 
of field of said first-mentioned and/or said further 3D reconstruction. 

10 

28. A method as claimed in any preceding claim wherein the angle subtended by a 
pair of correlated features at the corresponding feature of the object is 90 degrees ± 
30 degrees. 

15 

29. A method as claimed in claim 28 wherein the angle subtended by a pair of 
correlated features at the corresponding feature of the object is 90 degrees ±10 
degrees. 

20 30. Image processing apparatus for deriving a 3D representation of at least part of an 
object from overlapping 2D images thereof acquired from different viewpoints and 
correlated features of the respective 2D images in a region of overlap which are 
derived from a feature of the object common to the respective 2D images, the 

2 5 apparatus comprising reconstruction means arranged to form a 3D reconstruction in 
which said common feature of the object is located in a simulated 3D space in 
dependence upon both the mutual offset between said correlated features and a 
scaling variable. 

30 31. Apparatus as claimed in claim 30 comprising display means arranged to display 
a view of the reconstruction in simulated 3D space, the size of the displayed 
reconstruction being dependent upon the value of th scaling variable . 

32. Apparatus as claimed in claim 30 or claim 31 wherein said features are points. 

35 

33. Apparatus as claimed in claim 30 or 31 wherein said features are lines. 



34. Apparatus as claimed in claim 30 or claim 31 wherein said features are areas. 
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35. Apparatus as claimed in any of claims 30 to 34 which is arranged to calculate the 
viewpoints from the mutual offset between at least two but not all derivable pairs of 
correlated features in an overlapping region of the 2D images,to derive further pairs 

5 of correlated features from a search along epipolar lines derived from the viewpoint 
determination and to generate the 3D reconstruction of the object from the further 
pairs of correlated features and the viewpoints. 

36. Apparatus as claimed in claim 35 which is arranged to perform said calculation 
10 on fewer than one thousand pairs of correlated features. 

37. Apparatus as claimed in claim 36 which is arranged to perform said calculation 
on fewer than one hundred pairs of correlated features. 

15 

38. Apparatus as claimed in claim 37 which is arranged to perform said calculation 
on fewer than fifty pairs of correlated features* 

39. Apparatus as claimed in claim 38 which is arranged to perform said calculation 
20 on eight or fewer pairs of correlated points. 

40. Apparatus as claimed in claim 39 which is arranged to perform said calculation 
on two or three or four pairs of correlated points. 

25 41. Apparatus as claimed in any of claims 30 to 40 which is arranged to derive a 
further 3D reconstruction from further overlapping 2D images acquired from 
different viewpoints and correlated features of the respective further 2D images in a 
region of overlap which are derived from a feature of the object common to the 
respective further 2D images, the combining means being arranged to combine the 
first-mentioned 3D reconstruction and the further 3D reconstruction by 
manipulations in a simulated 3D space involving one or more of rotation and 
translation, the apparatus further comprising scaling means arranged to reduce or 
eliminate any remaining discrepancies between the 3D reconstructions by scaling 
35 one 3D reconstruction relative to the other along at least one axis. 
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42. Apparatus as claimed in claim 41 which is arranged to display both 3D 
reconstructions simultaneously and to manipulate them in simulated 3D space in 
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response to commands entered by a user. 

43. Apparatus as claimed in claim 42 which is arranged to perform the 
manipulations of the 3D reconstructions under the control of a computer pointing 
device. 

44. Apparatus as claimed in any of claims 41 to 43 which is arranged to correct 
distortion of said first-mentioned and/or said further 3D reconstruction resulting 
from an incorrect or incomplete calculation of a said viewpoint 

45. Apparatus as claimed in any of claims 41 to 44 which is arranged to correct the 
curvature of field of said first-mentioned and/or said further 3D reconstruction 
resulting from an incorrect or incomplete calculation of a said viewpoint. 

46. Image processing apparatus as claimed in any of claims 30 to 45, further 
comprising a camera whose position and/or orientation are not fully constrained 
with respect to the frame of the object, the camera being arranged to acquire said 2D 
images. 

47. Apparatus according to claim 46 comprisiing inertial sensor means arranged to 
determine the orientation of the camera relative to the object at the time of 
acquisition of said 2D images. 

48. Apparatus for deriving a 3D representation of at least part of an object, 
substantially as described hereinabove with reference to Figures 1 to 4 and 12 to 16 
15 of the accompanying drawings, optionally in conjunction with Figures 5, 6 and 7 
or any of Figures 8 to 1 1. 

49. A method of deriving a 3D representation of at least part of an object, 
substantially as described hereinabove with reference to Figures 1 to 4 and 12 to 16 
15 of the accompanying drawings, optionally in conjunction with Figures 5, 6 and 7 
or any of Figures 8 to 1 1 . 

50. A method of determining the motion of a camera relative to an object in the field 
of view of the camera comprising the steps of projecting the paths of features of the 
image of the object to a common vanishing point and determining the orientation of 
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the optical center of the camera lens in relation to this vanishing point. 

51. Apparatus for determining the motion of a camera relative to an object in the 
field of view of the camera comprising means for projecting the paths of features of 

5 the image of the object to a common vanishing point and means for determining the 
orientation of the optical center of the camera lens in relation to this vanishing point. 

52. A method of generating a 3D reconstruction of an object comprising the steps of 
projecting images of the object acquired by mutually aligned cameras into simulated 

10 3D space from aligned virtual projectors, the separation of the virtual projectors 
being variable by the user. 

53. Apparatus for generating a 3D reconstruction of an object comprising two 
aligned virtual projector means arranged to project images of the object acquired by 

15 mutually aligned cameras into simulated 3D space, the separation of the virtual 
projectors being variable by the user. 
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Abstract 

3D shape ac quisition apparatus 

Apparatus for deriving a reconstruction (30) of an object (3) from first and second 
imges comprises a camera (1) movable freely between two viewpoints (1, 1'). The 
camera orientations are arranged to be parallel at the instant of capture of the 
images, eg by superimposing the camera image plane at the second viewpoint on the 
camera image plane at the first viewpoint and tilting the camera about the X, Y and 
Z axes until the lines (LI to L3) connecting correlated points of the first image and 
the new image seen by the camera converge at a common "vanishing point" (VP). 
The line (ST) joining the camera positions is derived and a partial 3D reconstruction 
is derived by aligned virtual projectors (PR1, PR2/PR2') located an arbitrary 
distance apart (in dependence on a parameter entered by the user) on this line. The 
process is repeated from other pairs of viewpoints and the resulting partial 3D 
reconstructions scaled and combined on screen. 
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